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Abstact 

Background: Development and evaluation of the PEQ-CAMHS Outpatients, a parent completed questionnaire to 
measure experiences of outpatient child and adolescent mental health services (CAMHS) in Norway. 

Methods: Literature review, parent interviews, pre-testing and a national survey of 17,080 parents of children who 
received care at one of the 86 outpatient CAMHS in Norway in 2006. Telephone interviews were conducted with a 
random sample of non-respondents. Levels of missing data, factor structure, internal consistency and construct 
validity were assessed. 

Results: 7,906 (46.0%) parents or primary caregivers responded to the questionnaire. Low levels of missing data 
suggest that the PEQ-CAMHS is acceptable. The questionnaire includes three scales supported by the results of 
factor analysis: relationship with health personnel (8 items), information and participation (4 items), and outcome (3 
items). Item-total correlations were all above 0.6 and Cronbach's alpha correlations ranged from 0.88-0.94. The 
results of comparisons of scale scores with several variables relating to global satisfaction, outcome, cooperation, 
information, involvement and waiting time support the construct validity of the instrument. 

Conclusions: The PEQ-CAMHS Outpatients questionnaire includes important aspects of outpatient CAMHS from 
the perspective of the parent. It has evidence for data quality, internal consistency and validity and is 
recommended in surveys of parent experiences of these services. Future research should assess test-retest reliability 
and further tests of construct validity that include clinical data are recommended. 



Background 

The widespread acceptance of the importance of the 
views of service users in the assessment of health care 
quality has meant that the views of parents are increas- 
ingly sought in evaluations of the quality of care of child 
and adolescent mental health services (CAMHS) [1,2]. 
Compared with the patient satisfaction literature more 
generally, [3] there have been relatively few studies of 
user experiences with CAHMS but there is some evi- 
dence for small to moderate levels of correlation 
between satisfaction and mental health outcomes for 
children [4-7]. 
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Parents have responsibility for contacting health ser- 
vices on behalf of their child and their involvement in 
treatment is important for its success [8-10]. Moreover, 
parent experiences of care might influence their expec- 
tations of and involvement in future care [1,11]. Hence, 
parents can make an important contribution in the eva- 
luation of mental health services quality for children 
and parent-completed questionnaires have had applica- 
tions across a wide range of evaluations that include 
medication [12,13], community-based services [1,14,15], 
inpatient care [7,11] and health care plans [16]. 

Reviews of the literature have identified up to 28 
instruments and questionnaires designed to measure 
parent satisfaction with CAMHS [9,17]. These question- 
naires typically comprise multidimensional scales that 
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assess different aspects of parent satisfaction or experi- 
ences with such services. However, some questionnaires 
comprise global items while others sum to form a single 
score that might assess one aspect of parent satisfaction 
or general satisfaction [18]. Global items have been criti- 
cised for not providing adequately detailed information 
[9,11,19]. Multidimensional instruments provide more 
detailed information relating to specific aspects of care 
such as communication and information that can more 
fully inform quality improvement initiatives. 

Parent satisfaction questionnaires have been criticised 
more generally for not being comprehensive [20] and 
for lacking psychometric evidence including reliability 
and validity [7,9,20]. The former criticism relates to con- 
tent validity, a qualitative consideration as to whether an 
instrument or questionnaire adequately covers impor- 
tant aspects of parent satisfaction and in sufficient 
depth. An alternative approach involves asking users to 
rate their experiences of aspects of health care which 
may include communication and information provision, 
[21] and the collection of more objective information 
relating to whether specific health care events occurred. 
There is an inbuilt assumption that the aspects of 
experience covered by such questionnaires are related to 
user satisfaction and tests of construct validity often 
involve comparisons with global or single item measures 
of satisfaction [21]. It is important that users are 
involved in the development of such questionnaires to 
ensure that the most relevant aspects of the health care 
experience are included which lends the questionnaire 
content validity. 

The Parent Experiences Questionnaire for Outpatient 
Child and Adolescent Mental Health Services (PEQ- 
CAMHS Outpatients) was developed following a review 
of the literature that identified shortcomings with exist- 
ing questionnaires including a lack of content relevance 
to outpatient clinics in Norway. The questionnaire was 
designed for application in a national survey of parents 
or primary caregivers with children under 16 years of 
age attending mental health services outpatient centres 
throughout Norway [22]. The aim of the present study 
was to assess the data quality, internal consistency relia- 
bility and validity of the questionnaire. 

Methods 

Questionnaire development 

Questionnaire development and evaluation followed cri- 
teria that have been defined as important in patient 
satisfaction measurement [23]. The development of the 
questionnaire was based on a review of the literature 
that included existing parent satisfaction questionnaires, 
interviews with parents and consultation with an expert 
group comprising nine health professionals, a represen- 
tative from a user organisation and researchers. This 



process was designed to ensure content validity; that the 
questionnaire addresses important aspects of parent 
experiences in sufficient detail. 

Semi-structured interviews were undertaken with par- 
ents of eleven children under 16 years of age that had 
received care from two outpatient CAMHS in South 
East Norway. Following the interviews and any neces- 
sary revisions, the expert group was consulted. The 
questionnaire was then piloted by means of a postal 
questionnaire sent to 271 parents of children under 16 
years of age that had attended one outpatient CAMHS 
in South East Norway. The expert group was consulted 
again following the pilot survey. 

Data collection 

The questionnaire was included in national survey of 
17,871 parents of children aged under 16 years who had 
at least one appointment at one of the 86 outpatient 
centres in Norway between 1 September and 31 Decem- 
ber 2006 [22]. Patients were recruited using administra- 
tive data from the clinics and there was a maximum of 
400 users per clinic who were randomly selected if the 
number exceeded 400 during the recruitment period. 
This calculation was based on the number of respon- 
dents needed to compare the 86 clinics after taking 
account of non-response and the need for adjustment of 
covariates. 

Parents were sent a postal questionnaire following the 
transfer of administrative data from the clinics. The cov- 
ering letter informed them that the survey related to par- 
ent experiences. No instruction was given about which 
parent should complete the questionnaire and a question 
was included about who actually responded. Parents gave 
their informed consent to take part by ticking a box on 
the front page of the questionnaire which was returned 
in a reply paid envelope. Up to two reminders were sent 
to non-respondents at three and six weeks. 

Telephone interviews were conducted with a sample 
of 400 non-respondents following the second reminder; 
20 centres were randomly selected from the 86 centres 
in Norway and 25 patients were randomly selected for 
each centre. The interviews included four items that 
were found to explain the largest component of varia- 
tion in parent experiences following the pilot survey 
along with background questions. These four questions 
were summed to form a summary score which had 
satisfactory item-total correlations and a Cronbach's 
alpha of 0.84. The two groups of parents were compared 
using the t-test for the summary score and chi-square 
and Mann- Whitney U tests for categorical variables and 
ordinal variables. 

The Norwegian Regional Committee for Medical 
Research Ethics, the Data Inspectorate and the Norwe- 
gian Board of Health approved the survey. 
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Statistical analysis 

Items with ten percent or more missing data were con- 
sidered for removal from the final questionnaire. Princi- 
pal axis factoring with promax rotation was used to 
assess the underlying structure of the questionnaire [24]. 
Factors with eigenvalues above 1.0 were assessed for 
face validity and factor loadings greater than 0.4 were 
considered important. Items with important factor load- 
ings on more than one factor were considered for 
removal. Internal consistency was assessed by item-total 
correlation and Cronbach's alpha. The former measures 
the strength of association between an item and the 
remainder of its scale and it was expected that they 
would exceed 0.4 which is considered acceptable [25]. 
The latter assesses the overall correlation between items 
within a scale. For a scale to be considered sufficiently 
reliable for use in groups of patients, an alpha value of 
0.7 is considered acceptable [25,26]. 

In the absence of a gold standard measure of parent 
experiences, construct validity was assessed through 
comparisons of scale scores with responses to the addi- 
tional questionnaire items not included in the scales 
that related to various aspects of care. Following find- 
ings from the parent satisfaction literature, [27,28] it 
was hypothesised that moderate to high levels of corre- 
lation in the range 0.5 to 0.7, would be found between 
scale scores and parents overall satisfaction both in rela- 
tion to their child's treatment and their own experi- 
ences. There is some evidence for a low to moderate 
association between parent satisfaction and treatment 
outcomes for mental health services [4-7], and correla- 
tions in the range 0.3 to 0.6 were hypothesised for the 
perceived benefits of treatment accruing to the child 
and parent. Similar levels of correlation were hypothe- 
sised for responses to two items relating to health pro- 
fessionals cooperation, both within the clinic and with 
other professionals working outside the clinic including 
general practitioners and school teachers. 

The active participation of parents and families in 
mental health care of children and adolescents leads to 
better outcomes including parent satisfaction [10,14] 
and parent satisfaction was found to have low to moder- 
ate correlations with a measure of empowerment [10]. 
Hence, levels of correlation in the range 0.3 to 0.6 were 
expected between the scales and two items relating to 
parent understanding of treatment and involvement in 
dialogue with the clinic. Information giving is also posi- 
tively related to user satisfaction with health services [3] 
and it was hypothesised that scale scores would have 
low to moderate correlations in the range 0.3 to 0.6 
with perceptions of the adequacy of information relating 
to side-effects of their child's medication. 

There is evidence that parents are more satisfied with 
shorter waiting times [14,28] and it was hypothesised 



that scores would have low levels of correlation in the 
range 0.2 to 0.3 with how long parents had to wait for 
treatment for their children. Parent satisfaction ratings 
have been found to be positively associated with the 
time spent in treatment with mental health services 
[19,29,30] and hence it was hypothesised that scores 
would have low to moderate correlations in the range 
0.3 to 0.6 with parent perceptions of the adequacy of 
the number of hours spent at the clinic, number of 
clinic visits and the ease with which they were able to 
make contact with health professionals outside of con- 
sultation times when necessary. 

Unwanted behaviour by health professionals has been 
found to be consistently associated with user satisfaction 
more generally [3] and following previous findings from 
user satisfaction with mental health services [31], low 
correlations in the range 0.2 to 0.3 were expected with 
responses to two items relating to whether health pro- 
fessionals talked down to either the child or their 
parent. 

Results 

Questionnaire development 

Following the literature review and consultation with the 
expert group 28 questions were devised that covered 
accessibility, cooperation, information, overall satisfac- 
tion, outcomes and relationship with health personnel. 
These questions were included in the semi-structured 
interviews with the parents of eleven children. In general 
the questions were found to be acceptable and relevant. 
Some revisions were made following the interviews and 
consultation with the expert group. 

Of the 271 parents sent the pilot questionnaire, 122 
(45.0%) responded. The mean age of respondents was 
40.83 (sd 6.67) and 93 (76.2%) were the mother. Items 
had low levels of missing data and factor analysis of 14 
items that were relevant to all patients identified three 
scales of relationship with health personnel, information 
and participation and outcome. Following consultation 
with the expert group some small changes were made to 
the wording of some of the items prior to the main sur- 
vey and an additional item relating to "how the child 
functions outside of the family" was included as part of 
the outcome scale. Hence the PEQ-CAMHS Outpatients 
questionnaire included 15 items assessing parent experi- 
ences with five-point descriptive scales that assess various 
aspects of parent experiences of relevance to all parents. 

Additional items relating to information on medica- 
tion information and cooperation between health profes- 
sionals that were not relevant to all parents' included a 
"not applicable" option and other additional single items 
covered accessibility, overall outcome, overall satisfac- 
tion and unwanted behaviour. Finally, the questionnaire 
included several sociodemographic questions. 
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Data collection 

Of the 17,871 parents mailed the national survey ques- 
tionnaire, 7,906 responded; 755 did not have a correct 
address and 36 had cancelled their CAMHS appoint- 
ment which gives an adjusted response rate of 46.3%. 
Table 1 shows their characteristics. The mean age of the 
children was 11.3 (SD = 3.2) years and 2,946 (37.3%) 
were female. Of the 400 non-respondents conducted by 
telephone, 225 (56.3%) agreed to be interviewed. Com- 
pared to respondents to the main survey, there were 
three statistically significant differences that were impor- 
tant. Interviewees had a significantly different number of 
previous contacts with the clinic with 21.0% of intervie- 
wees and 14.4% of respondents reporting more than one 
previous contact. Interviewees also had a significantly 
different education level with 25.3% having received 
higher education compared with 44.3% for respondents. 
The parent experiences summary scores were not signif- 
icantly different. However, interviewees had slightly 
poorer experiences relating to their "viewpoints being 
taken seriously". There was also a significant difference 
for the number of clinic visits in the past three months. 
There were a large number of interviewees that had not 
visited the outpatient clinic in the past three months 
which follows from the interviews taking place after the 
second reminder. 

Statistical analysis 

The levels of missing data and descriptive statistics are 
shown in Table 2. All items met the criterion for miss- 
ing data and none were removed. Missing data ranged 
from 1.5% to 5.6% for items relating to "being met with 
politeness and respect" and "outcomes associated child's 
function outside of the family" respectively. These two 
items also had the highest and lowest scores of 4.35 and 
2.39 respectively. The majority of items had mean scores 
in the range of 3 to 4 on the scale of 1 to 5 where 5 is 
the best possible experiences of care. 

Factor analysis gave three factors accounting for 74.2% 
of the total variation (Table 2). Factor loadings were 
acceptable with all 15 items having loadings above 0.5 
and loadings below 0.3 on the other two factors. In 
order of the total amount of variation that they explain, 
the factors can be described as relationship with health 
personnel, information and participation, and outcome. 
Table 2 shows that the item-total correlations and Cron- 
bach's alpha for the final scales were acceptable, ranging 
from 0.65 to 0.89 and from 0.88 to 0.94, respectively. 

Table 2 shows the mean scores for the final 15-item 
questionnaire on a scale of 0 to 100 where 100 is the 
best possible experiences of care. Scores were skewed 
towards positive experiences of care and ranged from 
59.6 to 74.9 for the scales of information and participa- 
tion and relationship with health personnel respectively. 



Table 1 Child and parent characteristics for the main 
survey (n = 7,906) and non-respondents who were 
telephoned 3 (n = 225) 

Variable 



Main survey 
N (%) 



Telephone 
interviews N (%) 



Sex of patient 
Female 
Male 

Mean (sd) age of patient 

Sex of parent completing 
questionnaire 

Female 

Male 

Mean (sd) age of parent 
Education level of parent 

Primary school 

High school 

University graduate 

University postgraduate 
First language of parent 

Norwegian 

Sami 

Other Scandinavian 

Other European 

Not European 

Main activity of parent in the 
past week 

Work 

Sick leave 

Education 

Home worker 

Unemployed 

No. previous contacts with 
clinic 

0 



>1 

No. clinic visits in past three 
months 



2-5 

6-12 

12- 

Mean (sd) parent clinic 
knowledge 

Very poor 

Fairly poor 

Neither poor nor good 
Fairly good 
Very good 
Child badly treated? 
No 
Once 

Sometimes 



2946 (37.3) 
4947(627) 
11.32 (3.22) 



6515 (85.3) 
1124 (14.7) 
40.38 (6.87) 

827 (10.8) 
3456 (44.9) 
2049 (26.6) 

1361 (17.7) 

7292 (94.7) 
20 (0.3) 
87 (1.1) 
150 (1.9) 
149 (1.9) 



5203 (69.8) 
1322 (17.7) 
332 (4.5) 
453 (6.1) 
149 (2.0) 



5674 (74.2) 
877 (1 1.5) 
1100 (14.4) 



0 

2520 (35.2) 
3255 (45.5) 
1224 (17.1) 
153 (2.1) 
3.79 (1.04) 

321 (4.3) 
513 (6.8) 
1582 (21.0) 
3141 (41.7) 
1 968 (26.2) 

7310 (95.6) 
188 (2.5) 
128 (1.7) 



82 (36.4) 
143 (63.6) 
11.15 (3.39) 



1 73 (84.4) 
34 (1 5.6) 
39.88 (7.13) 

45 (20.4)** 
120 (54.3) 
39 (1 7.6) 
17 (7.7) 

214 (96.0) 

2 (0.9) 

3 (1.3) 
0 

4 (1.8) 



119 (55.3) 
61 (28.4) 

9 (4.2) 
20 (9.3) 

6 (2.8) 



1 56 (72.9)*' 
13 (6.1) 
45 (21.0) 



71 (32.7)** 
37 (17.1) 
73 (33.6) 
29 (13.4) 
7 (3.2) 

3.76 (1.03) 

5 (2.3) 
1 8 (8.4) 
62 (28.8) 
69 (32.1) 
61 (28.4) 

218 (99.1) 
1 (0.5) 
1 (0.5) 
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Table 1 Child and parent characteristics for the main sur- 
vey (n = 7,906) and non-respondents who were telepho- 
ned 3 (n = 225) (Continued) 



Many times 


23 (0.3) 


0 


Mean (sd) parent experiences 


65.29 (21.13) 


64.10 (21.10) 


scores b 






Care and consideration - 


2.81 (0.93) 


2.71 (0.94) 


parent c 






Took viewpoints seriously 


3.04 (0.91) 


2.91 (0.86)** 


Information - treatment 


2.11 (1.12) 


2.19 (1.12) 


Overall outcome - parent 


2.46 (1.09) 


2.51 (1.09) 



a T-test - age, parent experiences scores; Mann-Whitney test - parent 
experiences items, parent clinic knowledge; Chi-Square test - sex, education 
level, first language, main activity, number of previous contacts, number of 
clinic visits, child badly treated. 



Asterisks denote statistically significant differences between the two groups: 
*p < 0.05;**p < 0.01. 

b Sum of the four items transformed to 0-100 where 100 is the best possible 
experiences of care. 

c ltems are scored from 0-4 where 4 is the best possible experiences of care. 

The results of validity testing are shown in Table 3. 
The results of the 45 correlations were all highly signifi- 
cant. The three PEQ-CAMHS scores had moderate levels 
of correlation with overall satisfaction for how the child 
and parent were treated, the highest correlations being 
for the PEQ-CAMHS scale of relationship with health 
personnel. Slightly higher correlations were found for the 
child and parent outcomes, the highest being for the 
PEQ-CAMHS scores of outcome and relationship with 
health personnel. Moderate to high correlations were 
found for the two items relating to cooperation and the 
PEQ-CAMHS scores for relationship to health profes- 
sional. Low to moderate correlations were found for the 
two items relating to parent involvement, the highest 
being for the PEQ-CAMHS scores for relationship with 
health personnel and information/participation. The item 
relating to information associated with side effects of 
medication had the highest correlation with PEQ- 
CAMHS scores for information and participation at 0.60. 
As expected the correlations for the variables relating to 
availability and waiting time had lower correlations, the 
highest being for PEQ-CAMHS scores of relationship 
with health personnel. Finally, correlations of a low level 
were found for responses to the two items relating to 
whether the health personnel had talked down to them 
or their child, the highest being for PEQ-CAMHS scores 
for relationship with health professional. 

Discussion 

This study was part of the first national survey underta- 
ken in Norway to assess parent satisfaction with all Nor- 
wegian outpatient CAHMS. The development of the 
PEQ-CAHMS Outpatients followed a review of the lit- 
erature including existing questionnaires, interviews 
with parents, and consultation with an expert group of 



professionals involved in the delivery of CAMHS and 
researchers experienced in questionnaire and survey 
design. This was designed to ensure that the question- 
naire covered the important aspects of outpatient care 
in sufficient detail and hence had content validity. Par- 
ents are important participants in the delivery of mental 
health care to their children [9] and are not only well 
placed to judge service quality but can also inform the 
process of development and hence the content of ques- 
tionnaires that are designed to assess quality of care 
from their perspective. The resulting questionnaire 
underwent a rigorous process of piloting and testing for 
data quality, reliability and validity that have been 
recommended for evaluating such questionnaires [23]. 

The 15 items within the PEQ-CAMHS Outpatients 
contribute to three scales of relationship with health 
personnel, information and participation, and outcome. 
The levels of missing data in the range 1.5 to 5.6% sug- 
gest that it is acceptable to parents. 

The results of factor analysis and tests of internal con- 
sistency were strong empirical support for the three 
scales. The questionnaire has evidence for construct 
validity following the application of hypotheses based on 
previous research findings and theory. In this national 
survey the questionnaire scores were approximately nor- 
mally distributed in the range of 59.6 to 74.9 which 
means that there is potential for improvement in parent 
experiences over time and particularly for the scale of 
information and participation. 

The test-retest reliability of the questionnaire was not 
assessed due to limited resources. Previous surveys 
undertaken in Norway have found acceptable levels of 
test-retest reliability for other questionnaires that assess 
user experiences [21,27]. Test-retest reliability will be 
considered for a subsample of parents taking part in a 
future national survey of user experiences of outpatient 
CAMHS in Norway. The tests of construct validity were 
based on other questionnaire items completed by the 
parents and hence did not include objective or clinical 
data. Further tests of construct validity should include 
comparisons with administrative and clinical variables 
available from the clinics. 

The 15 items forming the three scales can be used 
alone if brevity is an important issue. Having just 15 
items makes it more suitable for inclusion in a longer 
questionnaire alongside other instruments that are 
designed to assess the outcomes of care for example 
within a randomised controlled trial and other forms of 
evaluative study. The additional items included within 
the questionnaire that related to other aspects of care 
including information on medication and cooperation 
between health professionals, should also be considered 
for inclusion in future surveys. Some of these items 
were not relevant to all parents but were considered 
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Table 2 Descriptive statistics, factor analysis and internal consistency 



Scale/item 


Missing 

(%) 


Mean 
(sd) 


1 


Frequency (%) 
2 3 4 


5 a 


Factor 
loading 


Cronbach's alpha (sco/e)/item- 
total correlation 


Relationship with health 


1 72 (2.2) 


74.92 














0.94 


personnel 




\ i o.m) 
















Care and consideration - 


170 (2.2) 


3.81 


148 


501 


1813 


3438 


1800 


0.82 


0.82 


parent 




!C\ Q31 


f\ Ql 

\ i .yj 


(6.5) 


(23.5) 


(44.6) 


(15 A) 






Understanding of situation 


256 (3.3) 


3.85 


162 


45 (5.9) 


1701 


3372 


1927 


0.84 


0.84 






rn qa\ 


ii 1 1 

(Z. 1 ) 




(22.3) 


(44.3) 


31 

(Z5.5) 






Care and consideration - child 


273 (3.5) 


4.14 


49 


216 


1110 


3431 


2791 


0.82 


0.78 






rn q?i 


rn ^1 


(2.8) 


(14.6) 


(45.2) 


71 






Met with politeness and 


1 16 (1.5) 


4.35 


49 


90 (1.2) 


616 


3359 


3640 


0.91 


0.78 


ros poet 




rn 731 

W-/ 5) 


rn 61 




(7.9) 


(43.3) 


(46 9) 






Spoke in a way that was 


242 (3.1) 


4.33 


30 


97 (1.3) 


585 


3529 


3387 


0.74 


0.66 


understandable 




id 71 1 


(Ci Al 




(7.7) 


(46.3) 


IAA A\ 






Took viewpoints seriously 


274 (3.5) 


4.04 


125 


321 


1275 


3251 


2624 


0.82 


0.83 






(u.y i j 


M £1 
( 1 .0} 


(4.2) 


(16.8) 


(42.8) 


(55.5) 






Had enough time for contact/ 


258 (3.3) 


3.90 


26 1 


977 


2122 


2865 


1387 


0.54 


0.69 


conversation 




fn 071 

(o.y/j 


13 .4) 


(12.8) 


(27.9) 


(37.6) 


f\ Q 11 

( 1 8.2) 






Cooperation with parent 


194 (2.5) 


3.54 


1 80 


489 


1457 


3333 


2217 


0.76 


0.86 






( I .U4J 


n ^i 

(1.5) 


(6.4) 


(19.0) 


(43.4) 


nc oi 






Information and participation 


273 (3.5) 


59.56 














0.88 






(15.1 1) 
















Participation in choice of 


323 (4.1) 


3.59 


39-1 


776 


1921 


2911 


1545 


0.89 


0.79 


treatment 




/i nQ\ 
( 1 .UBj 


(5.1) 


(10.3) 


(25.5) 


(38.6) 


nn ^i 
(ZV.5) 






Inflluence in choice of 


392 (5.0) 


3.39 


■-I91 


1058 


2225 


2492 


1212 


0.9 1 


0.76 


treatment 




( 1 .22) 


(b.6) 


(14.1) 


(29.8) 


(33.3) 


n a 11 
1 1 b.2) 






Information - treatment 


376 (4.8) 


3.1 1 


668 


1502 


2478 


2002 


844 


0.75 


0.75 






(I.I2J 


/ Q Ol 


(20.0) 


(33.1) 


(26.7) 


/1 1 31 
U -5) 






MiuniiauuM LMMU s lumuilium 


36^ Id f,) 


3 43 


449 


1018 


2117 


2631 


1 290 


0 53 


0 65 






(1.10) 


(6.0) 


(13.6) 


(28.2) 


(35.1) 


(17.2) 






Outcome d 


367 (4.6) 


72.08 














0.91 






(21.32) 
















Child's outcome from 


324 (4.1) 


3.96 


110 


202 


2187 


2394 


2653 


0.90 


0.84 


treatment 




(0.94) 


(1.5) 


(2.7) 


(29.0) 


(31.7) 


(35.2) 






Child's function in the family 


382 (4.9) 


3.81 


107 


175 


2574 


2366 


2266 


0.89 


0.86 






(0.93) 


(1-4) 


(2.2) 


(32.7) 


(30.1) 


(28.8) 






Child's function outside of 


441 (5.6) 


2.39 


128 


239 


2557 


2469 


2036 


0.83 


0.89 


family 




(0.92) 


(1.7) 


(3.2) 


(34.4) 


(33.2) 


(27.4) 







a 1 and 5 represent the worst and best possible patient experiences respectively. 

b Items within these scales have five-point descriptive scales from "not at all" to "to a very large extent". 



c The three scales are scored from 0 to 100 where 100 represents the best possible experiences. 
d Items within the outcome scale have a five-point scale from "much worse" to "much better". 



important enough for inclusion in the postal survey by 
the expert group and hence have content validity. 

The questionnaire is being used in national surveys of 
parent experiences that have included 86 outpatient 
clinics across Norway and the results for the 2006 sur- 
vey have been reported [22]. The results are included in 
the Norwegian national system of quality indicators and 
are designed to inform parent choice and quality 
improvement. They are available to parents and clinics 
in electronic and report form [22]. 

The questionnaire is designed for use with parents of 
children attending CAMHS outpatient clinics and while 
aspects of care that it assesses are of general relevance 



to CAMHS including the relationship with health per- 
sonnel, it does not include aspects of inpatient care pro- 
vision. The development of a questionnaire that 
included all aspects of CAMHS care would have neces- 
sitated a larger study that also included parents whose 
children had received inpatient care, both as a basis for 
informing the content of the questionnaire and also in 
testing. Some important aspects of parent experiences 
may be relevant for both inpatient and outpatient care 
and hence a generic questionnaire may be feasible. 
However, outpatient- and inpatient-specific modules will 
have greater content validity from a user perspective 
and are necessary for a more detailed understanding of 
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Table 3 Correlations between questionnaire scores and responses to individual questions 



Variable 



Relationship with health personnel Information and participation Outcome 



Overall satisfaction 3 : 

Treatment for child 

How parent was treated 
Overall outcome b : 

For child 

For parent 
Cooperation c : 

Among health professionals at the clinic 

Between the clinic and others involved with the child 
Parent involvement: 

Parent understanding of treatment from the clinic d 

Parent involvement in dialogue with the clinic e 
Talked down to by personnel: 

Child 

Parent 
Time and availability: 

Waiting time f 

Acceptable number of hours with clinic 9 
Number of clinic visits in the last three months h 
Contact with health professionals out of appointment time c 
nformation - side effects of medication c 



0.62 
0.64 

0.58 
0.71 

0.73 
0.62 

0.51 
0.26 

0.25 
0.41 

0.26 
0.45 
0.18 
0.50 
0.47 



0.49 
0.47 

0.55 
0.65 

0.65 
0.56 

0.56 
0.24 

0.15 
0.25 

0.20 
0.35 
0.04 
0.42 
0.60 



0.37 
0.28 

0.62 
0.52 

0.40 
0.30 

0.33 
0.09 

0.11 
0.15 

0.17 
0.26 
0.03 
0.26 
0.37 



All correlations are significant (p < 0.01). 

a Five-point scale: "very dissatisfied" to "very satisfied". 

b Five-point scale: "no benefit" to "a very large benefit". 

c Five-point scale: "Not at all" to "to a very large extent". 

d Five-point scaie: "very poor" to "very good". 

e Three-point scale: "never", "yes now and again", "yes often". 

f Four-point scaie: "no", "yes but not for long", "yes fairly long", "yes very long". 

9 Three-point scale: "very few hours", "too few hours", "enough hours". 

h Four-point scale: "only once", "2-5 times", "6-12 times", "more than 12 times". 



the quality of CAMHS care. Furthermore, the PEQ- 
CAMHS Outpatients is designed for parent completion 
and hence does not include the views of children and 
adolescents. Future research should also seek to include 
the views of children and compare these with parent 
experiences and clinical data including outcomes. 

The response rate of 46% lies within the range of 8 to 
63% found following a review of surveys of parent satis- 
faction with child and adolescent mental health; the 
median response rate to postal surveys was 33% [17]. 
More recent postal surveys lie in the range 22% to 52% 
for a survey without reminders [16] and with a tele- 
phone reminder [9], respectively. 'Personal contact' or 
more direct approaches which include surveying parents 
as part of initial and follow-up procedures at CAMHS 
produce higher response rates [17] but there is greater 
potential for social desirability bias [32,33]. Parents may 
report higher levels of satisfaction because they feel that 
this will be more acceptable to those administering the 
survey. Non-response causes bias if non-respondents 
systematically differ from respondents in relation to 



important study variables, here parent experiences and 
satisfaction. 

The findings for other follow-up surveys of Norwegian 
user experiences have shown that the response rates 
have not caused serious bias [33-36]. The response rate 
for the telephone survey was 56% following three postal 
contacts. There was no difference in overall parent 
experiences for respondents and telephone interviewees 
as assessed by a summary score comprising four items. 
However, there was one small but significant difference 
for one of these four items relating to "viewpoints being 
taken seriously". There was also a difference in the 
number of previous contacts with the clinic and educa- 
tion levels of parents. Interviewees reported a greater 
number of previous contacts and a smaller proportion 
had received higher education. The former may imply 
that non-respondents are parents of children with more 
severe conditions but this requires further investigation. 

Parent satisfaction questionnaires have been criticised 
for lacking evidence relating to quantitative aspects of 
psychometric testing including reliability and validity 
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[1,9,17]. The evaluation of the PEQ-CAMHS included 
assessing data quality, factor analysis to assess dimen- 
sionality, internal consistency and construct validity that 
have been recommended in the development of ques- 
tionnaires designed to measure user experiences and 
satisfaction [23]. Few studies have undertaken such a 
comprehensive evaluation which followed consideration 
of content validity; that the questionnaire includes the 
important aspects of parent experiences with outpatient 
CAMHS. Moreover data quality, an important indicator 
of the acceptability of items, has rarely been reported in 
previous studies. 

Conclusions 

The PEQ-CAMHS Outpatients includes important 
aspects of parent experiences with CAMHS that are 
based on the views of parents. The questionnaire has 
evidence for data quality, internal consistency, content 
validity and construct validity. The PEQ-CAMHS Out- 
patients is recommended for future applications 
designed to assess parent experiences of outpatient 
CAMHS and is being used in national surveys of parent 
experiences in Norway as a quality indicator. Test-retest 
reliability and further tests of construct validity, includ- 
ing comparisons with clinical variables are recom- 
mended for future research. 

Acknowledgements and funding 

The authors thank Tomislav Dimoski at the Norwegian Knowledge Centre 
for Health Services, Oslo, Norway for his contributions including developing 
software necessary for obtaining data from the CAMHS, conducting the data 
collection and quality assurance of data. We also thank Saga H0gheim who 
carried out most of the practical tasks relating to data collection, and 
members of the expert group that contributed to the development of the 
questionnaire and survey design. The survey was funded by the Norwegian 
Department of Health. 

Author details 

'Norwegian Knowledge Centre for the Health Services, Oslo, Norway. 
2 Centre for Child and Adolescent Mental Health, Eastern and Southern 
Norway, Oslo, Norway, department of Research and Development, Division 
of Mental Health Services, Akershus University Hospital, Lorenskog, Norway. 

Authors' contributions 

All authors contributed to the design of the questionnaire and survey. AMG 
conducted the analysis and drafted the manuscript. OAB and OH were 
involved in data acquisition. All authors have made significant contributions 
by critically reviewing the manuscript and have read and approved the final 
version. 

Competing interests 

The authors declare that they have no competing interests. 

Received: 3 December 2010 Accepted: 21 May 2011 
Published: 21 May 2011 

References 

1. Riley S, Stromberg A, Clark J: Assessing parental satisfaction with 
children's mental health services with the youth services survey for 
families. J Child Fam Stud 2005, 14:87-99. 



2. Garland AF, Haine RA, Boxmeyer CL: Determinates of youth and parent 
satisfaction in usual care psychotherapy. Evaluation and Program Planning 
2007, 30:45-54. 

3. Crow R, Gage H, Hampsom S, Hart J, Kimber A, Storey L, Thomas H: The 
measurement of satisfaction with healthcare: implications for practice 
from a systematic review of the literature. Health Technol Assess 2002, 
6(32). 

4. Shapiro JP, Welker CJ, Jacobson BJ: The Youth Client Satisfaction 
Questionnaire: development, construct validation, and factor structure. 

J Clin Child Psychol 1997, 26:87-98. 

5. Lambert W, Salzer MS, Bickman L: Clinical outcome, consumer satisfaction, 
and ad hoc ratings of improvement in children's mental health. J Consult 
Clin Psychol 1998, 66:270-9. 

6. Gerkensmeyer J: Patient satisfaction with mental health services: 
relationship to outcomes. In The 14th Annual Research Conference 
Proceedings: A System of Care for Children's Mental Health. Expanding the 
Research Base. Edited by: Liberton C, Kutash K, Friedman R. Tampa, Fla: 
University of South Florida; 2001:259-266. 

7. Kaplan S, Busner J, Chibnall J, Kang G: Consumer Satisfaction at a Child 
and Adolescent State Psychiatric Hospital. Psychiatr Serv 2001, 52:202-6. 

8. Delaney KR, Engles-Scianna B: Parents' perceptions of their child's 
emotional illness and psychiatric treatment needs. Journal of Child and 
Adolescent Psychiatric Nursing 1996, 9:15-24. 

9. Gerkensmeyer JE, Austin JK: Development and testing of a scale 
measuring parent satisfaction with staff interactions. J Behav Health Serv 
Res 2005, 32:61-73. 

10. Resendez MG, Quist RM, Matshazi DGM: A longitudinal analysis of family 
empowerment and client outcomes. J Child Fam Stud 2000, 9:449-60. 

11. Blader JC: Longitudinal Assessment of parental satisfaction with 
children's psychiatric hospitalization. Adm Policy Ment Health 2007, 
34:108-15. 

12. Swanson JM, Hechtman L: Using long-acting stimulants: does it change 
ADHD treatment outcome? Can Child Adolesc Psychiatr Rev 2005, 14(Suppl 
1)2-3. 

13. Steele M, Weiss M, Swanson J, Wang J, Prinzo RS, Binder CE: A randomized, 
controlled effectiveness trial of OROS-methylphenidate compared to 
usual care with immediate-release methylphenidate in attention deficit- 
hyperactivity disorder. Can J Clin Pharmacol 2006, 13:e50-62. 

14. Martin SJ, Petr GC, Kapp SA: Consumer satisfaction with children's mental 
health services. Child Adolesc Soc Work J 2003, 20:21 1-26. 

15. Day C, Davis H: The effectiveness and quality of routine child and 
adolescent mental health care outreach clinics. Br J Clin Psychol 2006, 
45:439-52. 

16. Heflinger CA, Simpkins CG, Scholle SH, Kelleher KJ: Parent/caregiver 
satisfaction with their child ' s medicaid plan and behavioral health 
providers. Ment Health Serv Res 2004, 6:23-32. 

17. Young SC, Nicholson J, Davis M: An overview of issues in research on 
consumer satisfaction with child and adolescent mental health services. 
J Child Fam Stud 1995, 4:219-38. 

18. Attkisson CC, Greenfield TK: The Client Satisfaction Questionnaire (CSQ) 
Scale and the Service Satisfaction Scale-30 (SSS-30). In Outcomes 
assessment in clinical practice. Edited by: Sederer LL, Dickey B. Baltimore: 
Williams 1996:120-127. 

19. Rey JM, Plapp JM, Simpson PL: Parental satisfaction and outcome: a 4- 
year study in a child and adolescent mental health service. Australian 
and New Zealand Journal of Psychiatry 1 999, 33:22-28. 

20. Ayton AK, Mooney MP, Sillifant K, Fowls J, Rasool H: The development of 
the child and adolescent versions of the Verona Service Satisfaction 
Scale (CAMHSSS). Soc Psychiatr Epidemiol 2007, 42:892-901. 

21. Garratt AM, Andersen Bjertnses 0, Krogstad U, Gulbransen P: The 
OutPatient Experiences Questionnaire: reliability and validity in 52 
Norwegian hospitals. Quality and Safety in Health Care 2005, 14:433-37. 

22. Bjertnass 0A, Garratt A, Helgeland J, Holmboe O, Dahle KA, Hanssen- 
Bauer K, R0ttingen JA: Parent assessment of outpatient child and 
adolescent mental health services. Tidsskr Nor faegeforen 2008, 
128:1041-1045. 

23. Sitzia J: How valid and reliable are patient satisfaction data? An analysis 
of 195 studies. Int J Qual Health Care 1999, 1 1:319-328. 

24. Hair J, Anderson RE, Tatham RL, Black WC: Multivariate data analysis with 
readings. 4 edition. Englewood Cliffs, New Jersey: Prentice-Hal! international 
Inc; 1995. 



Garratt et al. Child and Adolescent Psychiatry and Mental Health 201 1, 5:18 
http://www.capmh.eom/content/5/1 /1 8 



Page 9 of 9 



25. Nunnally JC, Bernstein IH: Psychometric theory. 3 edition. New York: 
McGraw-Hill; 1994. 

26. Kline P: The handbook of psychological testing London: Routledge; 1993. 

27. Garratt AM, Andresen Bjertnaes 0A, Barlinn JK: Parent Experiences of 
Paediatric Care (PEPC) questonnaire: reliability and validity following a 
national survey. Acta Paediatrica 2007, 96:246-252. 

28. Stallard P: Validity and reliability of the Parent Satisfaction Questionnaire. 
Br J Clin Psychol 1996, 35:311-8. 

29. Godley S, Fiedler E, Funk R: Consumer satisfaction of parents and their 
children with child/adolescent mental health services. Eval Program Plann 
1998, 21:31-45. 

30. Garland AF, Aarons GA, Saltzman MD, Kruse Ml: Correlates of adolescents ' 
satisfaction with mental health services. Ment Health Serv Res 2000, 
2:127-39. 

31. Garratt AM, Bj0rngard JH, Dahle KA, Bjertnaes 0A, Saunes IS, Ruud T: 
Psychiatric Out-Patient Experiences Questionnaire: data quality, reliability 
and validity in patients attending 90 Norwegian clinics. Nordic Journal of 
Psychiatry 2006, 60:89-96. 

32. Sitzia J, Wood N: Response rate in patient satisfaction research: an 
analysis of 210 published studies. Int J Qual Health Care 1998, 10:31 1-317. 

33. Bjertnass 0A, Garratt AM, Johannessen JO: Data collection methods and 
results in users surveys in mental health care. Tidsskr Nor Laegeforen 2006, 
126:1481-1483. 

34. Guldvog B, Hofoss D, Pettersen Kl, Ebbesen J, Ronning OM: PS-RESKVA 
-(Patient Satisfaction, Results and Quality - patient satisfaction in 
hospitals). Tidsskr Nor Laegeforen 1998, 118:386-391. 

35. Bjertnaes OA, Garratt A, Botten G: Nonresponse bias and cost- 
effectiveness in a Norwegian survey of family physicians. Eval Health Prof 

2008, 31:65-80. 

36. Bjertnaes OA, Garratt A, Iversen H, Ruud T: The association between GP 
and patient ratings of quality of care at outpatient clinics. Earn Pract 

2009, 26:384-90. 



doi:1 0.1186/1 753-2000-5-1 8 

Cite this article as: Garratt et al,: Parent experiences questionnaire for 
outpatient child and adolescent mental health services (PEQ-CAMHS 
Outpatients): reliability and validity following a national survey. Child 
and Adolescent Psychiatry and Mental Health 201 1 5:18. 



Submit your next manuscript to BioMed Central 
and take full advantage of: 

• Convenient online submission 

• Thorough peer review 

• No space constraints or color figure charges 

• Immediate publication on acceptance 

• Inclusion in PubMed, CAS, Scopus and Google Scholar 

• Research which is freely available for redistribution 



Submit your manuscript at S~\ RioM _j Central 

www.biomedcentral.com/submit \ J BHMWea central 



